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ABSTRACT 



This paper discusses the lack of effective assessment of 
student learning in graduate programs and presents the results of a case 
study designed to improve the assessment of graduate student learning at 
North Dakota State University (NDSU) . Graduate faculty in several departments 
at NDSU decided to adapt Primary Traits Analysis (PTA) to graduate tasks in 
their master's and doctoral programs. PTA involves identifying the essential 
characteristics of successfully completed tasks or projects. Faculty began by 
specifying three to six student outcomes for each graduate program at each 
degree level. Departments then developed check sheets for use by faculty in 
assessing student performance on traits identified for the task. The faculty 
involved have concluded that PTA can be adapted for use by graduate faculty 
and that evidence for student assessment can be gathered that goes beyond 
"pass" and is useful in improvement efforts. (Contains 13 references.) (MDM) 
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Assessment of Student Learning in Graduate Programs 
Grades of at least "B," "pass” on comprehensive exams, and "satisfactory" on the 
thesis or dissertation are traditional hallmarks of graduate education. The students who 
receive a graduate degree have successfully “passed" all the tasks in the program. 

A review of the literature indicates little has been presented at professional meetings 
or published about the assessment of learning by graduate students. In their review, 
Kaylor and Johnson (1994) observed that graduate and professional education are 
seldom addressed in discussions of outcomes assessment. Examination of assessment 
plans and reports reveals some disturbing tales about what is going on. 

The following statement is from the assessment plan of a prestigious midwestern 
university which shall remain unnamed, "The goal of assessment of graduate student 
learning in 1994 was to evaluate the services and staff of the graduate office. Students 
were asked to respond to a ten-item questionnaire in which a rating was given to such 
matters as the courtesy, promptness, and knowledge of the staff, and the quality and 
usefulness of printed materials." Another example, "The knowledge and skills of 
graduate students are assessed continuously and include standards of admission, 
retention, and graduation . . ." What does this tell you about the academic achievement 
of students in the program when the emphasis is on IMPROVING student learning? 

When the goal of assessment is to improve student learning, data that students 
successfully completed the requirements (thesis, oral seminar, defense, etc.) do not 
provide evidence to identify areas of program strength or areas where the program and 
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learning could be improved. Faculty in graduate programs agree that they "know it when 
they see it," and use their expertise and experience to judge appropriate graduate level 
performance. There is little attention paid to student outcomes (Conrad & Egan, 1990). 

Efforts to assist graduate faculty in re-examining their practices in graduate 
education have met with only modest success. Getting beyond the notion that since the 
student successfully completed all the requirements for the degree "the learning is 
assured," to gathering data relative to what was actually learned is progressing slowly. 
While alumni surveys, exit interviews of student satisfaction with the program, and other 
indirect measures provide useful information, they do not provide the direct evidence of 
what was learned and what was not learned so well. Graduate faculty, assessment 
directors, and AIR professionals with assessment responsibilities are searching for ways 
to use the activities of graduate programs for more direct assessment of learning. 

When graduate faculty are approached with the question, "Do all graduate students 
do equally well on all aspects of the graduate program?" their response is, "No." Thus 
the intent is to find ways acceptable to graduate faculty to gather such evidence. One 
computer science professor described it this way, "How do we know whether the 
students are actually learning what we want them to learn?" 

The traditional judgment about students’ progress by the professor is not what this is 
about. Professors will continue to use various means, such as projects, papers, 
examinations, and the like, to "grade” students' progress through the program. Grades 
in courses, for example, do not provide the answers. The assumption is made that 
students who get A’s have learned more than students who get C's but there is no 
evidence of precisely what either group learned well or less well. Whatever the 
assessment method used, it must give the department the ability to diagnose 
program/student strengths and weaknesses so that constructive improvement measures 
can be devised where needed. Every method used should provide evidence of outcome 
attainment including specific areas of effective or ineffective performance. Evidence is 
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needed of learning in relation to the outcomes; evidence of achievement across 
students to improve programs and learning. 

A case study reporting the results of faculty efforts at North Dakota State University 
(NDSU) to use the traditional activities of graduate programs is reported. Briefly, NDSU 
has about 9500 students. There are about 550 graduate students enrolled in 40 
master’s programs and 18 Ph.D. programs. 

The Method 

Graduate faculty in several departments at NDSU decided to adapt Primary Traits 
Scoring analysis to graduate tasks in their master’s and doctoral programs. 

Primary Traits Analysis comes originally from the scoring of written papers by 
English teachers. (A list of references on the technique and its uses is provided.) The 
technique was first used on a large scale by the National Assessment of Educational 
Progress in 1 974. The purpose was to provide information about what students know 
and can do. 

Primary Traits Analysis involves identifying the essential characteristics of a 
successfully completed task or product. What traits or characteristics are inherent in the 
task or product? It involves the faculty making explicit the things they look for in the 
comprehensive exam responses, the research proposal, the oral seminar presentation, 
the thesis, or the oral defense. The criteria being used are explicitly stated. For 
example, what constitutes a thesis that ”passes"? Traits are the things the faculty care 
about. 

Faculty began by specifying three to six student learning outcomes for each 
graduate program at each degree level. Faculty recognized that the Graduate School's 
forms for program completion were too global to be useful in providing evidence of 
attainment of the outcomes. They wanted to retain the traditional activities of graduate 
programs, only make them more useful for improvement. 
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Primary traits analysis was already being tried in some undergraduate programs so 
they decided to try it with graduate program activities. The faculty had already agreed 
they "knew it when they saw it" so they began by identifying the important 
characteristics, or traits, of successfully completed tasks, such as the review of literature, 
the oral seminar, the thesis or dissertation, and the final oral defense. 

Results 

The departments have developed check sheets for use by faculty in assessing 
student performance on the traits identified for the task. Usually a three-level rating 
scale is used. Student performance is then rated on the various tasks. It takes very little 
additional time of the faculty. The Graduate School receives the "pass" or "satisfactory" 
it requires. The department accumulates the data across students to identify areas of 
strength and areas where learning may need improvement, although each individual 
student may have performed well enough to "pass." 

The Zoology faculty have identified a further concern from their use of primary traits 
analysis on the final oral defense. 

Conclusion 

The faculty’s purpose was to find a feasible way to use the usual tasks in graduate 
programs to gather evidence of student learning in order to improve that learning. They 
have concluded that Primary Traits Analysis can be adapted for use by graduate faculty. 
Evidence can be gathered that goes beyond "pass" and is useful in improvement efforts. 
A concern remains with the problems of having only a small number of program 
completers in some master’s and doctoral programs. 
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